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[57] ABSTRACT 

An apparatus for monitoring the position of multiple objects 
in a space including a target supervisor, a location 'supervi- 
sor, and a hazard monitoring supervisor. The tracking super- 
visor receives target data from a sensor, characterizes and 
tracks selected objects, and provides a target output having 
multiple features respective of the selected objects. The 
location supervisor characterizes and displays multiple fea- 
tures in the space, and provides a location output having the 
aforementioned features therein. The hazard monitoring 
supervisor detects and responds to a predetermined hazard 
condition, and provides a detectable notice of such hazard 
condition, responsive to the target output and the location 
output A data logger for selectively retains the target output, 
the location output, or both. The position monitoring appa- 
ratus has six modes: full operation, non-airport-surveillance- 
radar, isolation, set-up, adaptation, and simulation. An inte- 
grator mode is included wherein the target supervisor 
receives data from a surface detection radar, an airborne 
surveillance radar, a secondary surveillance radar, a global 
positioning-system based sensor, a ground-based sensor, or 
an auxiliary sensor. 



33 Claims, 4 Drawing Sheets 
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AIRPORT INTEGRATED HAZARD 
RESPONSE APPARATUS 



BACKGROUND OF THE INVENTION 

This application is related to co-pending patent applica- 
tion Ser. No. 08/494,118, filed Jun. 23, 1995, entitled 
"Aircraft Landing Determination Apparatus and Method," 
which is incorporated herein by reference and which is 
assigned to the same assignee hereof. 

1 . Field of the Invention 

The invention herein regards air traffic control systems, 
and particularly to air traffic control systems integrating 
airborne and ground-based target tracking and providing an 15 
integrated hazard response. 

2. Description of the Prior Art 

Accidents involving aircraft and ground vehicles and 
aircraft ground collisions can occur at busy airports. More 
than three times as many near-accidents occur on the ground 20 
as in the air. In one such accident, an aircraft strayed onto the 
wrong runway and was struck by a second aircraft, resulting 
in a loss of life. In another accident, two aircraft collided 
when one aircraft was cleared to land on an occupied 
runway. Continuous situation monitoring in fog, snow, at 25 
night, and under other adverse conditions, can cause fatigue 
in human controllers which may lead to a collision between 
objects on airport surfaces. 

Medium-range airport surveillance radar, such as the 
Automated Radar Terminal System (ARTS) is good for 30 
detecting and tracking many aircraft within a large volume 
of airspace. However, such systems do not provide adequate 
surveillance coverage for ground-resident objects, including 
aircraft that are in the taxiing, holding (stopped), takeoff or 
landing phases of their flight profiles. 

Airport surface detection equipment (ASDE) systems can 
provide high-resolution, short-range, clutter-free, surveil- 
lance information on aircraft and ground vehicles, both 
moving and fixed, located on or near the surface of airport 4Q 
movement and holding areas under all weather and visibility 
conditions. An ASDE system formats incoming surface 
detection radar information for a desired coverage area, and_ 
presents -it to local and ground controllers on high-resolu- 
tion, bright displays in the airport control tower cab. 45 

However, an ASDE system will be effective only if the 
controllers are observing the radar display, which may be 
unlikely during good visibility conditions. The increasing 
sophistication of electronic equipment in air traffic control 
towers can result in an increasing number of computer 50 
displays and alarms. The large number of displays and 
keyboards in the tower cab can result in a cramped and 
cluttered environment which may be relieved by an inte- 
grated display apparatus. In addition to ASDE and ARTS 
systems, other sensor systems such as, for example, second- 55 
ary surveillance radar (SSR), and global positioning system 
(GPS) can provide logically disparate parameters in physi- 
cally disparate locations within the tower cab, further adding 
to the controllers' burden. The lack of integration between 
sensor systems in the tower cab, including ARTS, ASDE, go 
SSR, GPS, and the like, can yield gaps in controller aware- 
ness of the unfolding situation. 

\In addition, a large, busy airport is an environment having 
a vast number of possible conflict situations. In such a 
dynamic environment, the potential for collision between 65 
any given aircraft and any one of possibly many ground- 
resident, and nearby airborne, objects may not be recognized 
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until it is too late to avoid the collision. Also, the occurrence 
of a conflict in one area of the airport may draw controllers' 
attention away from evolving conflict situations. 

What is needed is apparatus which can integrate inputs 
from disparate sensor systems to aid controllers in adverse 
conditions, such as poor visibility conditions or peak traffic 
periods, and provide an improved means of detecting and 
alerting the controller to pending surface conflict. 

SUMMARY OF THE INVENTION 

The invention herein provides an apparatus for monitor- 
ing the position of multiple objects in a predefined space 
including a target supervisor, a location supervisor, and a 
hazard monitoring supervisor. The tracking supervisor char- 
acterizes and tracks selected objects, receiving target data 
from at least one sensor. Hie target supervisor provides a 
target output having multiple predetermined features respec- 
tive of the selected objects. The location supervisor charac- 
terizes and displays multiple, predetermined features respec- 
tive of the predefined space, and provides a location output 
having the aforementioned predetermined features therein. 
The hazard monitoring supervisor detects and responds to at 
least one predetermined hazard condition. It provides a 
detectable notice of such hazard condition, responsive to the 
target output and the location output. The apparatus can also 
include a data logger for selectively retaining the target 
output, the location output, or both. 

The target supervisor can include a target collector for 
extracting at least one target track from the target data, and 
tracking it over a predetermined period of time. The target 
supervisor further can include a target manager for deter- 
mining the predetermined features respective of the selected 
objects responsive to the target data. The location supervisor 
can include an adaptation manager for providing selected 
portions of the predetermined features respective of the 
predefined space. The hazard monitoring supervisor can 
include a hazard avoidance manager which can respond to 
the predetermined hazard condition according to a prede- 
termined selectable set of hazard avoidance logic rules. The 
hazard monitoring supervisor can also include an alarm 
manager for determining a manner of presentation of the 
detectable. notice. In addition, the hazard monitoring super- 
visor can include a situation display manager for providing 
a visual display, an aural display, or a printed display, 
responsive to the predetermined hazard condition. The data 
logger can include an event logger as well as an adaptation 
logger. The target collector can include a first interface unit 
between a surface detection radar and the position monitor, 
and a second interface unit for connection between an 
airport surveillance radar and the position monitor. 

The position monitoring apparatus can have six modes: a 
full operation mode, a non-airport-surveillance-radar mode, 
an isolation mode, a set-up mode, an adaptation mode, and 
a simulation mode. It also can have an integrator mode 
wherein the target supervisor receives the target data from 
multiple sensors, at least one sensors being of a type 
different from a respective other. In this mode, the target 
supervisor can integrate the disparate target data to provide 
the target output thereby. Each of the sensors from which the 
target supervisor can receive data is from the group consist- 
ing of a surface detection radar, an airborne surveillance 
radar, a secondary surveillance radar, a global positioning- 
system based sensor, a ground-based sensor, and an auxil- 
iary sensor. 

Other details, objects, and advantages of the invention 
will become apparent as the following description of certain 
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present preferred embodiments thereof proceeds. . The 
accompanying drawings show presently preferred embodi- 
ments of the invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 5 

FIG. 1 is an illustration of the functional relationship 
between an ASDE system, an ARTS system, and the AIHR 
according to the invention herein. 

FIG. 2 is a block diagram of one embodiment of the *o 
AIHR. 

FIG. 3 is a illustration of one embodiment of the func- 
tional components of the AIHR. 

FIG. 4 is a detailed block diagram of the AIHR shown in 
FIG. 3. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 

The invention herein can provide an airport integrated 20 
hazard response apparatus (AIHR) and supply controllers 
with automatic conflict warnings and alerts, enabling them 
to act swiftly to reduce the incidence of collisions and other 
runway and taxiway accidents. The AIHR can analyze 
surface object movements for indications of possible threats, 25 
and can automatically alert controllers to inadequate vehicle 
spacing, inappropriate or unauthorized movements or posi- 
tioning within the airport area and its associated airspace, 
and even runway debris. Such objects can include aircraft, 
trucks, people, animals, and the like. The AIHR can track a 30 
preselected number of targets including aircraft on final 
approach, as well as those that are departing or landing, and 
objects, including aircraft, that are taxiing or stopped. Tar- 
gets may be represented as computer-generated symbols, 
improving display readability. 

The AIHR can provide the ASDE-3 or other surveillance 
display with an overlay of symbology on the display map 
coordinates covering the entire airport which may be 
zoomed, windowed, offset, and rotated. This information 40 
may also be remotely linked to displays at other locations 
including in aircraft or vehicles. By zooming in on a 
particular area of interest, controllers in the tower cab or - . 
other observers can see images of planes, ground vehicles, 
and people on the ground on the radar display consoles. The 45 
AIHR can provide position monitoring of a predefined space 
which can include airport surfaces at the ground and a 
preselected range of air space thereabove, for example, 200 
feet in altitude or 10 miles in range. 

. The AIHR can utilize information from a number of 50 
sources to develop current and projected locations of mov- 
ing objects compared to defined data surfaces. In addition to 
surface-based data from an ASDE system, such other 
sources may include, for example, nearby-airborne-target 
data from ARTS, tracking data from SSR, absolute spatial 55 
location data from GPS devices, and vectored location 
information from squittering systems. The AIHR provides 
fusion of disparate sensor data so that controllers can 
comprehend and respond appropriately to the nature and 
priority of a plurality of possible hazard situations. go 

Predicted and actual trajectories of each object are com- 
puted and compared with the predicted and actual trajecto- 
ries of all other objects in a predefined space, which space 
may include a ground surface and a preselected portion of air 
space thereabove. Based on information such as the move- 65 
ment state, previous locations, vehicle type and size, surface 
type, surface status, airport geometry, and the like, AIHR can 
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determine if each moving object exceeds allowable preset 
limits for each surface, movement state, and surface type 
and status with respect to other objects. It can also determine 
if objects remain stationary on surfaces beyond allowed 
preset limits. 

The surface of interest can be divided into predetermined 
segments so that object locations may be determined relative 
to those segments. Each segment may, in turn, be divided 
into a plurality of hazard logic cells, each of which may be 
analyzed for the existence of a potential conflict. A hazard 
avoidance manager can analyze the plurality of data inputs 
to classify an existing situation. The existence of the alert 
condition can be raised by a comparison of the actual 
situation with graphical data tables which may contain 
predefined rules such as, for example, spacing rules based on 
factors such as surface configuration and the object move- 
ment state. These alerts may be used to create voice and 
display messages or audio/visual alarms, and can control 
traffic lights, data lime: messages lb operators, and send 
information to remote master controllers or vehicle opera- 
tors. 

The AIHR can provide visual and aural alarms for runway 
incursions. Advisories for cautions and warnings involving 
aircraft, vehicles and obstacles on active runways, approach 
paths to runways, and intersecting taxiways can be provided. 
Visual alarms may be provided at each controller position 
with an ASDE-3 display. A common tower cab aural alarm 
can be provided via loud speaker in the tower cab. Map 
editors may be provided in the adaptation manager to define 
areas of interest, such as runways and intersecting taxiways 
for which valid sensor data is available. This information 
can be provided to the hazard monitoring supervisor to 
detect potentially hazardous situations. 

The AIHR can have a display processor which generates 
display information mixed with information provided to 
ASDE operational display units (ODU) to provide overlaid 
video images. In addition, the entry device for AIHR can be 
the ASDE-3 display control unit (DCU), or a local personal 
computer or computer workstation platform (collectively, 
PC platform). AIHR equipment may be kept in the tower 
equipment room except for a loud speaker and amplifier in 
the tower cab therefore reducing the amount-of. equipment 
requiredin the tower cab. AIHR can communicate with a 
plurality of ASDE display channels, for example, eight 
ASDE display channels. 

^pTIG. 1 illustrates the operation of AIHR 10 in conjunction 
with ASDE system 12 and ARTS system 14. ASDE system 
12 can be of the ASDE-3 type, or system 12 can be another 
surveillance system, or display system, or both. In general, 
the ASDE display processor interface control (DPIC) unit 16 
can recgivejata from radar transceivers 18a, 18fc. Each of 
ASDE controller work stations 2Qa-< can include monitor 
21a-c for data display and input devices 22a~c for data entry 
by controllers. Each of work stations 2Qa-c can be associ- 
ated with ASDE display processor unit 23^-c, respectively. 
DPIC 16 routes display videa,signals 24a-c to monitors 
2\a-c from the respective DPU 23a-c. Similarly, DPIC 16 
routes controller input receive through DCU link 2Sa~c to 
respective DPU 23a-c. Although not shown specifically in 
conjunction with DPU 23a and 23fc, the various signals 
communicated by DPIC 16 and a DPU are shown with 
regard to DPU 23c. For example, DPIC 16 and DPU 23c can 
share communication data along bidirection al commumca- 



tionHnk 26 L R adar data 27 are received by DPU 23c and 
processea uTprovide display video 28, which in turn, is 
routed to ASDE monitor 21c. Controller information input 
and display control signals are transmitted along DCU link 
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29 between DPIC 16 and DPU 23c. It is possible for AIHR 
10 to provide additional displ ay information to the control- 
lers by providing an addit^^ for process- 
ing by DPU 23c whlSi signal 30ismrxed with other data 
within DPU 23c to form a composite videojyLgna) which is 
transmitted by display vi deo signal 28 to ASDE monitor 21c 
or other display as commanded to the DPIC. Communica- 
tion between AIHR 10 and DPU 23c may be facilitated by 
bidirectional communication link 31. To provide the radar 
data necessary for hazard analysis and response, and video 
input to DPU 23a-c, radar signal 32 can be provided to 
AIHR 10 from DPIC 16. AIHR 10 can also accept data 
signal 33 from ARTS unit 14 so that radar returns from 
airborne aircraft can be fused in AIHR 10, processed for 
safety checks, ^tngsrnilteiUe DPU 23c via video signal 30 or 
both, and be routed to DPIC 16 over displaYv ideo signal 2 8 
for display on monitor 21c on workstation 20c. AIHR 10 can 
provide audible signals through audible alarm 33, which 
signals may include synthesized voice and tone signals. 
AIHR 10 can work with either ASDE system 12 or ARTS 
system 14, or both, and with othe rauxilia ry sensor systemsji 20 

FIG. 2 illustrates one embodiment of alTanport integrated 
response apparatus according to the invention herein. AIHR 
50 can collect and fuse data from disparate sensors such as, 
for example, ASDE system 51, ARTS system 52, GPS 
sensor 53, SSR sensor 54, and other sensor 55, sensors 53, 
54, and 55 being auxiliary sensors. As used herein, the term 
sensor is inclusive of a particular detector, such as a radar 
transceiver, and its associated processing equipment; one 
such sensor being ASDE system 51. Because the data output 
by the aforementioned sensors may be presented to AIHR 50 
at different data rates and in different physical and logical 
formats, interface units such as ASDE data processor (ADP) 
56, ARTS interface unit (AIU) 57, and auxiliary sensor 
interface 58 can be used to present data to main processing 
unit (MPU) 60 in a suitable format. Collectively, ADP 56 
and AIU 57 can be used, in part, to serve as target collector 
59. In addition to acting as an interface to ASDE system 51, 
ADP 56 can provide additional signal processing for target 
extraction. ADP 56 may also perform target centroiding and 
target tracking. Similarly, AIU 57 can filter target data from 
the airport surveillance radar (ASR), track ASR targets, and 
provide a bidirectional data link between ASDE system 51 
and ARTS system 52. MPU .60. can collect sensorinputs 
from and provide control and responsive outputs to ADP 56, 
AIU 57, and interface 58 by way of bidirectional data links 
64, 65, and 66, respectively. In addition to sensor fusion, 
MPU 60 can correlate ASR and ASDE track files, manage 
target tracking and provide audible alert generation includ- 
ing synthetic voice and audible tone messages using audible 
alarm 61. MPU 60 can also provide surface monitoring and 
situation analysis, employing a hazard monitoring supervi- 
sor. This supervisor can use selectable, predetermined haz- 
ard avoidance rules to determine the nature, probability, and 
priority of a plurality of possible hazard situations. A built-in 
self-test may also be incorporated into MPU 60. 

Display processor 62 can effect warning symbol genera- 
tion including hold bars, icons, symbol blinking, and the 
like. Display processor 62 also acts as the display interface 
between AIHR 50 and ASDE system 51. Processor 62 
formats input for display on ASDE DPU 63<3-c. MPU 60 can 
include a PC platform such as a 80486-based personal 
computer or workstation system, with ADP 56, AIU 57, and 
interface 58 being connected to the PC platform by way of 
a MICROBUS channel. 65 

The AIHR 50 can have six modes of operation. First, the 
full operating mode permits AIHR 50 to operate in conjunc- 
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tion with both the ASDE system 51 and ARTS system 52. 
This is because separate ASDE and ARTS data extractors 56, 
57, respectively, provide data to AIHR 50 for coordination 
of target data processing. 

Second, in the "non-ARTS" mode, the AIHR 50 can be 
capable of operation without ARTS system 52. In the event 
that there are no approaching aircraft, or ARTS unit 52 is not 
operational, the AIHR 50 can continue to process other input 
information, including ASDE returns, to monitor the posi- 
tion and movement of ground targets. 

Third, in the isolation mode, AIHR 50 may be operated 
separately from ASDE system 51 without affecting the 
operation of ASDE system 51. In this mode of operation, 
ASDE system 51 does not rely on AIHR 50 for inputs in data 
processing or display. 

Fourth, the adaptation mode provides a unique setup for 
each runway traffic flow and each access to each runway, 
where feasible. Menus can be provided for each runway to 
specify both configuration and status. 

Fifth, the setup mode specifies runway incursion logic 
parameters and system parameters. These parameters may 
be entered using the AIHR PC platform. In this mode, the 
tower staff can select a combination of runways and appli- 
cable traffic flows and parameters. The groupings of pre- 
defined runway configurations and associated definitions are 
buildable and selectable via keyboard to facilitate airport 
reconfiguration. 

Sixth, a simulation mode can be provided to perform 
engineering development and training off line using 
recorded or scenario data and injected targets via PC plat- 
form input devices. 

An integrator mode can also be implemented. In this 
mode, the target supervisor can integrate data from two or 
more disparate sensors such as, for example, surface detec- 
tion radar, airborne surveillance radar, secondary surveil- 
lance radar, and global positioning system-based sensors, 
ground-based sensors, and other auxiliary sensors to pro- 
duce the target output for the target tracker. 

The AIHR apparatus can accept and retain both physical 
and operational runway and taxiway definitions, data sepa- 
ration rules, system parameters, and display alert priority 
conditions. The runway/taxiway physical data may be set-up 
upon installation of the apparatus. Separation parameters 
may be derived for each airport during apparatus testing and 
validation. Operational configurations and system param- 
eters can be selectable. Alert priorities also can be selectable 
and may be programmed in software. Furthermore, adapta- 
tion runway data may be grouped into user selectable 
entities. 

FIG. 3 present a functional illustration of an AIHR 
according to one embodiment of the invention herein. AIHR 
100 provides an apparatus for monitoring the position of a 
plurality of objects in a predefined space having target 
supervisor 102 for characterizing and tracking a selected 
number of the objects, location supervisor 104 for charac- 
terizing and displaying predetermined features respective of 
the predefined space, and hazard monitoring supervisor 106 
for detecting at least one predetermined hazard condition 
responsive to target supervisor output 108 and location 
supervisor output 110, and for providing a detectable notice 
thereof. Hazard monitoring supervisor 106 can include situ- 
ation display manager 107 for producing and managing the 
detectable notice which may include visual, aural, and 
printed notice. " ^ 

FIG. 4 provides an illustration of AIHR 100 in FIG. 3 
respective of a physical layout. Managers 114, 116, 118, 120, 
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122, and 143, logger 126, and processor 140 may be 
contained within MPU 60. Target supervisor 102 in FIG. 3 
can include target collector 112 and target manager 114, seen 
in FIG. 4. Location supervisor 104 in FIG. 3 can include 
adaptation manager 116 and area definition manager 118, 
seen in FIG. 4. Hazard monitoring supervisor 106 in FIG. 3 
can include hazard avoidance manager 120 and alarm man- 
ager 122, seen in FIG. 4. AIHR 100 can also include data 
logger 126 which may include an event logger and an 
adaptation logger. 

Continuing in FIG. 4, target collector 112 can extract and 
track target information received from at least one of a 
surface detection- object detector 128, which can be for 
example, an ASDE-3 radar system, and an airborne surveil- 
lance detector 129 which can be, for example, an ARTS 
radar system. Target collector 112 may also provide for 
synthetic target generation, injection, and playback, accept- 
ing target data from simulator 109. 

Target collector 112 can extract targets of interest from 
input sensor data using target extractors 131, 133. Target 
extractor 131 can include first data interface 130 for extract- 
ing information from surface detection sensor 128, and 
target extractor 133 can have second interface 132 for 
receiving information from surveillance radar sensor 129. 
Together with the graphical areas defined by adaptation 
manager 116, AIHR 100 can acquire and track targets from 
surface detection and surveillance radar sensors 128, 129, 
respectively, on runways, taxiways, within intersections, and 
on approach paths. Extractors 131, 133 can use information 
such as airport geometry, target geometry, clutter map, and 
primary and secondary acquisition areas to isolate the targets 
of interest. 

Target trackers 134 can provide a peak amplitude smear 
from the ASDE radar data to target manager 114. Target 
manager 114 can have a clutter filter, centroider, and an 
alpha/beta tracker. The clutter filter provides raw extracted 
target data to the centroider. The centroider locates the 
centroid of the extracted image received from the clutter 
filter. The resultant image centroid can then be provided to 
the alpha/beta tracker which can be sensitive to both 
straight-line movement and target maneuvering, and which 
can provide a next-position estimate and velocity for the 
target thereby. - - 

Location supervisor 104 may also include an area defi- 
nition manager 118 to provide selected milieu information 
therein to target supervisor 102. Included in the milieu 
information can be the aforementioned clutter map, as well 
as selected system parameters, predefined constant tables, 
defined area parameters, and target tracking information. 
Such system parameters can include Pulse Repetition Fre- 
quency (PRF) data from radar, tower height, mode of 
operation, clutter update frequency, and scale factors. Tabu- 
lar information can include trigonometric coefficient tables, 
target ID table, and alpha/beta filter coefficient table. System 
definitions can include primary and secondary information 
acquisition areas, and clutter areas. Tracking information 
can include target status, position, and velocity, as well as 
selected other tracking parameters. 

Area definition manager 118 can also include airport 
configuration data, and movement/direction parameters in 
the milieu information. Airport configuration data can 
include "normal" surface directions, and airport surface 
types. Movement/direction parameter inputs can include 
velocity threshold ranges, and travel direction windows. 
Furthermore, area definition manager 118 can include infor- 
mation such as, for example, surface directions, surface 
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direction codes, surface boundary coordinates, surface inter- 
section coordinates, runway threshold coordinates, and seg- 
ment data. 

Target manager 114 can perform at least seven functions. 
First, it can maintain target reports from ASDE 128, the 
playback system, and a synthetic target injection device in 
target collector 112. Second, target manager 114 can gen- 
erate target states for use by other components of the system. 
Third, target manager 114 can compute spatio-temporal 
parameters such as target velocities and accelerations. 
Fourth, target manager 114 can determine the current loca- 
tion of targets on airport surfaces. Fifth, target manager 114. 
can predict the future location of targets on airport surfaces 
based on information including prior target reports, com- 
puted spatio-temporal parameters, and current target loca- 
tions. Sixth, target manager 114 can provide to display 
processor 136 target information, such as, for example, 
target identification, target type, and target position. Sev- 
enth, target manager 114 can detect target report anomalies. 

Target manager 114, when provided surveillance and 
surface detection radar information from the first and second 
interface units, determines directions of movement with 
respect to movement areas, and classifies targets as either 
stopped or in one of a predetermined number of movement 
states. In addition, target manager 114 can predict future 
target locations based on derivation of velocity and accel- 
eration information from previous position reports. In one 
embodiment, a second order fit of the previous five position 
reports can be used. 

Target manager 114 can maintain target reports and com- 
pute target-related parameters such as, for example, veloci- 
ties and accelerations. Target manager 114 can also generate 
target states which may be provided to hazard avoidance 
manager 120 for analysis. Also, target manager 114 can 
determine whether targets are on active runways and moving 
at speeds greater than or less than a selected speed param- 
eter. Movement states may be based in part on detected 
vehicle speeds. 

In general, target manager 114 analyzes and processes 
information regarding target location, direction, and move- 
ment. Target location can be with respect to a predefined 
.position within the selected space, which can include a 
particular surface in the airport configuration. Target direc- 
tion can be determined with respect toga particular surface 
direction. 

Target manager 114 can determine target movement by 
comparing the target's current state with its previous state 
relative to a movement parameter database to provide the 
target movement data of interest. Target data can include 
information from at least one sensor, target identification 
data, position coordinates, velocity vectors, acceleration, 
and target state. 

Target manager 114 can provide current and predicted 
target location information which includes the surface(s) on 
which each target is located as well as the surface type, 
which is based on the airport configuration. The current and 
predicted locations of each target can be provided by a 
particular spatial segment within the airport. In general, 
target direction can be used to determine whether the target 
is approaching an intersection, or other targets or objects. 
. Outputs from the target manager can include target loca- 
tion, target direction, and target movement state. Target 
location can include the particular surface number upon 
which the target is located. Target direction can be the states 
which include a normal direction, an opposite direction, a 
"in the direction of, a "not in the direction of, and general 
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direction, such as north, east, south, or west. Target move- 
ment states can include arrival, landing, departure, departure 
abort, taxi, and stop. 

Simulator 109 can provide synthetic physical objects and 
tracks, and operational situations including simulated hazard 5 
conditions to target supervisor 102, location supervisor 104, 
and hazard avoidance supervisor 106. Simulator 109 can be 
used, for example, to inject a selected number of objects, 
including debris, ground vehicles, aircraft, and the like, into 
the desired predefined space in the vicinity of the airport, for 10 
purposes such as routine and hazard-response training. 

Adaptation manager 116 can maintain the plurality of 
predetermined physical and operational features respective 
of a predefined space including, for example, spatial con- 
figurations of an airport at which the apparatus may be 15 
installed. The predetermined features can be maintained as 
graphical entities which, when used in conjunction with the 
target manager, can permit AIHR 100 to monitor targets on 
runways, taxiways, within intersections, on approach paths, 
and in the nearby airspace. Adaptation manager 116 can also 20 
maintain system parameters and support installation and 
operator data. 

Hazard monitoring supervisor 106 can detect at least one 
predetermined hazard condition responsive to target super- 
visor 102 and location supervisor 104, as seen in FIG. 3. 25 
Hazard monitoring supervisor 103 in FIG. 3 can include 
hazard avoidance manager 120 which analyzes information 
received from target manager 114 and from adaptation 
manager 116. Hazard avoidance manager 120 detects unde- 
sirable conditions, or hazards, such as, the incursion of an 30 
object into a selected portion of the predefined space which 
is allocated to another object, such as, for example, an 
aircraft. In addition, hazard avoidance manager 120 identi- 
fies and assigns a priority to an alert condition as provided 
by a preselected hazard rule set. 35 

Hazard avoidance manager 120 can use preselected 
parameters for analysis and detection of alert situations. 
Such parameters can include predetermined target separa- 
tion parameters, airport configuration, and airborne and ^ 
ground traffic situations. Separation parameters can include 
time separations, distance separations, times to intersections, 
and stop times on surfaces. The airport configuration can be 
with regard to runway/taxiway geometry, runway usage 
suchf as open/close, direction of movement, taxiway, and the 45 
like, and segment state such as, segment under repair, snow 
plowing, maintenance, and the like. 

Airborne traffic situation information can include infor- 
mation regarding approaching traffic such as an object's 
location and an aircraft's landing runway assignment. On 50 
the other hand, ground traffic situation can include informa- 
tion regarding moving and stopped traffic or obstructions on 
surfaces. This can include locating movement, providing 
movement direction, velocity, and acceleration, and the 
vehicle state. The spatio-temporal relationship between the 55 
two targets, respective of the targets' position in the moni- 
tored space can be maintained in a set of hazard avoidance 
logic tables. 

Hazard avoidance manager 120 can also provide target 
pair analysis by aetermining the relative positions of the 60 
target pair, and rjerforrning projection computations using 
the spatio-temporal parameters of the target pair to predic- 
tively select particular, predetermined projection situations 
which may be indicative of a hazard or alert situation. Such 
projection situations can include a "chase" situation, where 65 
two targets on a single surface are moving in the same 
direction, a "head-on" situation, where two targets on a 



single surface are moving towards each other, and a "cross- 
ing" situation, where two targets are converging upon each 
other at an intersection. A no-problem situation can exist 
when either none of the aforementioned situations exits, or 
if, after target pair analysis, the target movements are 
projected to avoid a hazard or alert situation. 

Using rules for intersections as delineated in the hazard 
avoidance logic tables in hazard avoidance manager 120, 
target manager 114 can determine whether a target is on a 
taxiway or runway which is intersecting with an active 
runway and is predicted to enter the taxiway or runway 
within a preselected number of seconds, determining if an 
alert is required. Hazard avoidance manager 114 can also 
assess the situation for landing an approaching aircraft to an 
active or inactive runway. 

1 Hazard monitoring supervisor in FIG. 3 can include alarm 
manager 122, seen in FIG. 4, which can maintain the 
aforementioned alert priorities, can generate video and aural 
commands, and control the utilization of the alerts. Alert 
notification can include the use of graphic indicators such as 
icons around "caution" targets, a flashing icon around 
"warning" targets, fixed hold bars, moving hold bars, and 
arts indicator lines. Alert notification may also include the 
provision of an alert message window required by hazard 
avoidance manager 120. Audio alerts can also include voic e 
*alerts/j \ ■ ' 

Alarm manager 122 can control the alert indications 
received from hazard avoidance manager 120 through pri- 
oritization, message scrolling, video display coordination, 
and repetitive alert suppression. With alert prioritization, 
messages may be displayed in priority order based on the 
alert type, the movement state and the direction of target 
movement in an alert message window. With selective 
message scrolling, the five highest priority messages will 
remain on the display in the alert message window. Mes- 
sages can remain on queue while the voice alarm alert is 
active. Those active messages which may have scrolled off 
of the message window can be redisplayed when the space 
is available. 

Under video display coordination, voice messages may 
not be interrupted and the display can contain the message. 
In addition, text messages can remain active on the display 
while an incursion persists, and for at least as long as the 
spoken voice alert is provided. Repetitive alerts can be 
suppressed where subsequent alerts for the same targets and 
same conditions may not be reissued. 

Alarm manager 122 can prioritize multiple alerts. For 
example, a warning has a higher priority than a caution. A 
situation involving two targets has a higher priority than one 
involving one target, for a single target conditions on the 
runway take priority over conditions on a taxiway. Vehicles 
moving in a direction opposite to what is considered "nor- 
mal" for the surface have a higher priority than vehicles 
moving in the normal direction. Departures, arrivals, and 
landings have higher priority than taxied or stopped 
vehicles. Also, new conditions have a higher priority than 
sequential alerts for the same target. Every message can be 
assigned an individual priority. 

In general, a message processor in alarm manager 122 
retrieves alert data from an alert queue, determines the 
classification and relative priority of the alert, and updates 
the display target status. Problem targets may be sent to 
display processor 136. Voice message and alert display 
messages may be generated with the message content based 
on alert condition and the associated surface areas involved. 
Text messages may also be sent to display processor 136. In 
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addition, if there are multiple messages, the message pro- 
cessor processes the next alert in the queue. Also, the 
message processor handles fresh alerts by cleaning up 
obsolete alerts. 

Situation display manager 107 in FIG. 3 can provide 5 
video and aural alert information to the air traffic controllers 
via ASDE DPUs 138. The visual and aural situation indicia 
can include, for example, icons, alarm messages, menus, 
ARTS indicator lines, and fixed and moving hold bars, and 
are provided to DPUs 138 as video signals which can be 10 
transmitted to the ASDE display unit and overlaid upon the 
ASDE video display. 

If an alert condition exists from an approaching aircraft, 
the alert condition is displayed on the approach to the active 
runway using an ARTS indicator line. The ARTS indicator 15 
line is displayed next to the runway image to indicate the 
distance from the runway threshold, for example, three 
miles. Alarm manager 122 can display live target symbols, 
alert messages, and airport configurations for both live and 
simulated situations on both the AIHR PC platform and the 20 
ASDE-3 displays by way of display processor 136. Alert 
message positioning on the display can be controlled by the 
operator. 

Hold bars and alerts may be used to indicate the advis- 
ability of entering an active runway. Each surface segment 25 
may be divided into the preselected number of hazard logic 
cells for which alerts may be generated or suppressed. A 
moving hold bar may be placed in front of any targets 
involved in situations for which such is specified in the 
hazard logic tables. Fixed hold bars may be displayed at 30 
intersections and removed from intersections when the tar- 
get passes them. Fixed hold bars can be activated along all 
intersections in front of landing, departing, departure abort 
aircraft, and approaching aircraft within a selectable dis- 
tance of the runway threshold. 

Situation display manager 107 in FIG. 3 may also contain 
print manager 143 and voice processor 140, seen in FIG. 4. 
Print manager 143 can print current adaptation parameters or 
alternate sets of airport configuration adaptation parameters ^ 
which are available for selection. Voice processor 140 can 
generate synthetic voice aural alerts which may be provided 
in the tower cab separately from the ASDE operational 
display units. Furthermore, print manager 143 manager can 



35 



45 



provide a hard copy of a particular situation by generating a 
printed report. 

AIHR 100 can also include data logger 126 which may 
retain situational information such as, the data pertinent to a 
particular event, and the data associated with a particular set 
of features respective of the monitored space, for example, 50 
adaptation data. Data logger 126 can have the capability to 
record all setup data including safety and system parameters, 
maps, configurations, and clutter maps. It can also record 
cautions, warnings, controller queues, and target data. This 
information can be time- tagged whenever an update in an 55 
item occurs. Other information such as adaptation condi- 
tions, events, and selective area filtering data can be recalled 
from storage by the operator for future use. 

While specific embodiments of practicing the invention 
have been described in detail, it will be appreciated by those 60 
skilled in the art that various modifications and alternatives 
to those details could be developed in light of the overall 
teachings of the disclosure. Accordingly, the particular 
arrangements disclosed are meant to be illustrative only and 
not limiting to the scope of the invention which is to be given 65 
the full breadth of the following claims, and any and all 
embodiments thereof. 



We claim; 

1. An apparatus for monitoring the position of a plurality 
of objects in a predefined space comprising: 

(a) a target supervisor for characterizing and tracking 
selected ones of said plurality of objects, said target 
supervisor receiving target data from at least one sen- 
sor, said target supervisor providing a target output 
having a first plurality of predetermined features 
respective of said selected ones; 

(b) a location supervisor for characterizing and displaying 
a second plurality of predetermined features respective 
of said predefined space, said location . supervisor pro- 
viding a location output having said second plurality of 
predetermined features therein; and 

(c) a hazard monitoring supervisor for detecting and 
responding to at least one predetermined hazard con- 
dition and providing a detectable notice thereof, said 
hazard monitoring supervisor being responsive to said 
target output and said location output. 

2. The apparatus of claim 1 further comprising a data 
logger for selectively retaining at least one of said target 
output and said location output. 

3. The apparatus of claim 1 wherein said target supervisor 
further comprises a target collector for extracting at least one 
target track from said target data, said target collector 
tracking said at least one target track over a predetermined 
period of time. . 

4. The apparatus of claim 1 wherein said target supervisor 
further comprises a target manager for determining said first 
plurality of predetermined features respective of said 
selected ones responsive to said target data. 

5. The apparatus of claim 1 wherein said characterizing 
and tracking selected ones by said target supervisor includes 
predicting at least one of said first plurality of predetermined 
features in said target output. 

6. The apparatus of claim 4 wherein said determining by 
said target manager includes predicting at least one of said 
first plurality of predetermined features respective of said 
selected ones responsive to said target data. 

7. The apparatus of claim 1 wherein said location super- 
visor includes an adaptation manager for providing selected 
portions of said second plurality of predetermined features 
respective of said predefined space. _ _ _ - 

8^ The apparatus of-claim 1 wherein said location super- 
visor includes an area definition manager for providing 
selected milieu information. 

9. The apparatus of claim 1 wherein said detecting and 
responding by hazard monitoring supervisor includes pro- 
jecting a first position of a first target, projecting a second 
position of a second target, comparing said first position 
with said second position, and determining a preselected 
projection situation from said comparing. 

10. The apparatus of claim 9 wherein said preselected 
projection situation includes one of a chase situation and a 
head-on situation and a crossing situation and a no-problem 
situation. 

11. The apparatus of claim 1 wherein said hazard moni- 
toring supervisor further comprises a hazard avoidance 
manager, said hazard avoidance manager responding to said 
at least one predetermined hazard condition according to a 
predetermined selectable set of hazard avoidance logic rules. 

12. The apparatus of claim 10 wherein said hazard moni- 
toring supervisor further comprises a hazard avoidance 
manager, said at least one predetermined hazard condition 
including said preselected projection situation, said hazard 
avoidance manager responding to said at least one prede- 
termined hazard condition according to a predetermined 
selectable set of hazard avoidance logic rules. 
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13. The apparatus of claim 1 wherein said hazard moni- 
toring supervisor further comprises an alarm manager for 
determining a manner of presentation of said detectable 
notice. 

14. The apparatus of claim 1 wherein said hazard moni- 5 
toring supervisor further comprises a situation display man- 
ager for providing at least one of a visual display and an 
aural display and a printed display, said providing being 
responsive to said at least one predetermined hazard condi- 
tion. w 

15. The apparatus of claim 2 wherein said data logger 
includes an event logger. 

16. The apparatus of claim 2 wherein said data logger 
includes an adaptation logger. 

17. The apparatus of claim 3 wherein said target collector 
further comprises a first interface unit between a surface 15 
detection radar and said target extractor and an second 
interface unit for connection between an airport surveillance 
radar and said target extractor. 

18. The apparatus of claim 1 having at least one of a full 
operation mode and a non-airport-surveillance-radar mode 20 
and an isolation mode and a set-up mode and an adaptation 
mode and a simulation mode. 

19. The apparatus of claim 1 having an integrator mode 
wherein said target supervisor receives said target data from 

a plurality of sensors, at least one sensor of said plurality of 25 
sensors being of a type different from a respective other of 
said plurality of sensors, said target supervisor integrating 
said target data to provide said target output thereby. 

20. The apparatus of claim 19 wherein said at least one 
sensor is from the group consisting of a surface detection 30 
radar and an airborne surveillance radar and a secondary 
surveillance radar and a global positioning-system based 
sensor and a ground-based sensor and an auxiliary sensor. 

21. The apparatus of claim 19 wherein said respective 
other is from the group consisting of a surface detection 
radar and an airborne surveillance radar and a secondary 35 
surveillance radar and a global positioning-system based 
sensor and a ground-based sensor and an auxiliary sensor. 

22. The apparatus of claim 1 further comprising a simu- 
lator for simulating the presence of a selected number of said 
plurality of objects in said predefined space, said simulator 4 ° 
being connected to said target supervisor, said location 
supervisor, and said hazard monitoring supervisor. 

23. An apparatus for monitoring the position of a plurality 
of objects in a predefined space comprising: _ 

(a) a target supervisor for characterizing and tracking a 45 
" selected number of said plurality of objects, said target 

supervisor receiving target data from at least one sen- 
sor, said target supervisor providing a target output 
having a first plurality of predetermined features 
respective of said selected ones, said target supervisor 50 
having a target collector and a target manager, said 
target collector for extracting at least one target track 
from said target data, said target collector tracking said 
at least one target track over a predetermined period of 
time, and said target manager for determining said first 55 
plurality of predetermined features respective of said 
selected ones responsive to said target data; 

(b) a location supervisor for characterizing and displaying 
a second plurality of predetermined features respective 
of said predefined space, said location supervisor pro- 
viding a location output, said location supervisor hav- 
ing an adaptation manager for providing selected por- 
tions of said second plurality to said location output, 
said location supervisor having an area definition man- 
ager for providing selected milieu information, said 
location supervisor being operably connected to said 65 
target supervisor, and 
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(c) a hazard monitoring supervisor for detecting and 
responding to at least one predetermined hazard con- 
dition and providing a detectable notice thereof, said 
hazard monitoring supervisor being operably con- 
nected to said target supervisor and said location super- 
visor, said hazard monitoring supervisor being respon- 
sive to said target output and said location output, said 
hazard monitoring supervisor having a hazard avoid- 
ance manager and an alarm manager and a situation 
display manager, said hazard avoidance manager 
responding to said at least one predetermined hazard 
condition according to a predetermined selectable set of 
hazard avoidance logic rules, said alarm manager for 
determining a manner of presentation of said detectable 
notice, said situation display manager for providing at 
least one of a visual display and an aural display and a 
printed display, said providing being responsive to said 
at least one predetermined hazard condition. 

24. The apparatus of claim 23 wherein said characterizing 
and tracking selected ones by said target supervisor includes 
predicting at least one of said first plurality of predetermined 
features in said target output. 

25. The apparatus of claim 23 further comprising a data 
logger for selectively retaining at least one of said first 
plurality of said target output and said second plurality of 
said location output, said data logger having an event logger 
and an adaptation logger. 

26. The apparatus of claim 23 having at least one of a full 
operation mode and a non-airport-surveillance-radar mode 
and an isolation mode and a set-up mode and an adaptation 
mode and a simulation mode. 

27. The apparatus of claim 23 further comprising a 
simulator for simulating the presence of a selected number 
of said plurality of objects in said predefined space, said 
simulator being connected to said target supervisor, said 
location supervisor, and said hazard monitoring supervisor. 

28. The apparatus of claim 23 having an integrator mode 
wherein said target supervisor receives said target data from 
a plurality of sensors, al least one sensor of said plurality of 
sensors being of a type different from a respective other of 
said plurality of sensors, said target supervisor integrating 
said target data to provide said target output thereby. 

29. The apparatus, of claim 28 wherein said at least one 
sensor irfrom the group consisting of a surface detection 
radar and an airborne surveillance radar and a secondary 
surveillance radar and a global positioning-system based 
sensor and a ground-based sensor and an auxiliary sensor. 

30. The apparatus of claim 29 wherein said respective 
other is from the group consisting of a surface detection 
radar and an airborne surveillance radar and a secondary 
surveillance radar and a global positioning-system based 
sensor and a ground-based sensor and an auxiliary sensor. 

31. The apparatus of claim 23 wherein said detecting and 
responding by hazard monitoring supervisor includes pro- 
jecting a first position of a first target, projecting a second 
position of a second target, comparing said first position 
with said second position, and determining a preselected 
projection situation from said comparing. 

32. The apparatus of claim 31 wherein said preselected 
projection situation includes one of a chase situation and a 
head-on situation and a crossing situation and a no-problem 
situation. 

33. The apparatus of claim 31 wherein said at least one 
predetermined hazard condition including said preselected 
projection situation. 

* * * * * 
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AUDIO/VIDEO SYSTEM WITH AUXILIARY 
DATA 

The present application is a continuation of co-pending 
patent application, Sezen et al., Ser. No. 60/049,078, filed 5 
Jun. 9, 1997. 

BACKGROUND OF THE INVENTION 

The present invention relates to a system for encoding, 
sending, and receiving audio and video that includes auxil- 10 
iary data. 

Digitally encoding video, consisting of multiple sequen- 
tial frames of images, is a common task that traditionally has 
been performed by dividing each frame into a set of pixels 
and defining a sample luminance value, and possibly a color 
sample value, for each pixel. The development of powerful 
inexpensive computer technology has made it cost effective 
to develop complex systems for encoding moving pictures to 
achieve significant data compression. This permits, for 2Q 
example, high definition television (HDTV) to be broadcast 
within a limited bandwidth. The international standard 
which has been chosen for the transmission of HDTV is the 
Moving Pictures Experts Group 2 (MPEG-2) standard. The 
MPEG-2 video compression standard is based on both intra 25 
and inter coding of video fields or frames and achieves 
compression by taking advantage of either spatial or tem- 
poral redundancy in the video sequence. 

An additional video encoding system has been proposed, 
designated as the MPEG-4 standard. MPEG-4 is a system of 30 
encoding video by which a moving picture composed of a 
sequence of sequential frames, generally accompanied by 
audio information, may be transmitted or otherwise sent 
from a moving picture sending device ("sender") to a 
moving picture receiving device ("receiver") in an efficient 35 
and flexible manner. Interactive applications are also antici- 
pated in the MPEG-4 standard, but for ease of description, 
this patent application describes the case in which a moving 
picture is sent from the sender to the receiver only. It is to 
be understood that this patent application includes the case ^ 
in which control information may be sent from receiver to 
sender. 

" MPEG-4 supports the transmission of a "Binary Format 
for Scene Description" (BIFS) which specifies the compo- 
sition of objects in a sequence of frames or fields represent- 45 
ing a three dimensional scene. In BIFS a scene is divided 
into a plurality of "scene elements." For example, as shown 
in FIG. 1A, a scene with a presenter, a blackboard, a desk, 
a globe, and an audio accompaniment is broken up into 
scene elements in panel 12, and reconstituted into a com- 50 
plete scene, with all the scene elements properly arranged in 
panel 14. In other words, each frame of a MPEG-4 based 
video system is composed of a plurality of scene elements 
that are arranged according to directives specified in the 
associated BIFS information. As such MPEG-4, similar to 55 
MPEG-2, is directed to a system for encoding and transmit- 
ting video and audio information for viewing a movie, albeit 
using a different protocol than MPEG-2. An example of a 
suitable systems decoder for MPEG-4 is shown in FIG. IB. 

Referring to FIG. 2, each scene element 20 is represented 60 
by an object to which is assigned a node 26 in a generally 
hierarchical tree structure (panel 16). The scene elements 
requiring input streaming data include an object descriptor 
14. The tree structure provides a convenient data structure 
for representing a sequence of video fields or frames. Each 65 
object descriptor in turn includes at least one elementary 
stream descriptor 28 which includes such information as the 
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data rate, location information regarding the associated data 
stream(s) and decoding requirements for its respective logi- 
cal channel (described below) for updating information 
regarding the data object. The data stream sent through a 
particular pipe for a particular object descriptor is generally 
referred to as an elementary stream. Such information may 
include, for example, data describing a change in shape, 
colorization, brightness, and location. Every elementary 
stream descriptor includes a decoder type (or equivalent 
stream type structure) which publishes the format or encod- 
ing algorithm used to represent the transmitted elementary 
stream data. FIGS. 3 and 4 show a simplified object descrip- 
tor . format and an elementary stream descriptor format, 
respectively. 

In an MPEG-4 system, both the sender and the receiver 
can use a respective "Delivery Multimedia Integration 
Framework" (DMIF) for assigning the data channels and 
time multiplexing scheme based on requests from the 
MPEG-4 applications. The DMIF, in effect, acts as an 
interface between an MPEG-4 application and the underly- 
ing transport layer, sometimes referred to herein as a "data 
pipe." The pipes generally refer to the physical or logical 
interconnection between a server and a client. As an object 
descriptor is sent from the sender to the receiver, the sender 
DMIF examines each of its elementary stream descriptors 28 
and assigns a pipe to each one based on the requirements 30, 
32, 34 and 36 included in the elementary stream descriptors 
28. For example, an elementary stream with a more exacting 
quality of stream requirement 34 would be assigned a pipe 
that would permit high quality transmission. The pipe 
assignment is sent to the receiving DMIF. For example, 
depending on the transmission system each pipe (logical 
channel) could be a different periodic time portion of a 
single connection; each pipe could be different connection 
through the internet; or each pipe could a be different 
telephone line. In general, DMIF establishes the connections 
and ensures that both the sender and the receiver are in 
synchronization for tying a particular pipe to a particular 
elementary stream at both the sending and the receiving end. 

The DMIF also assigns an association tag to each elemen- 
tary stream. This tag provides a single identification of each 
elementary stream from the sending DMIF to the receiving 
DMIF and vice versa. Accordingly, the association tag is a 
unique end-to-end DMIF pipe identifier. Any channel 
change or reassignment initiated by the sender or receiver 
(for an interactive system) would require a different asso- 
ciation tag. 

Referring to FIG. IB, DMIF delivers data to an MPEG-4 
application by way of "Access Unit Layer Protocol Data 
Units" (AL-PDUs). The AL-PDUs configuration descriptor 
26 in the elementary stream descriptor 28 (FIG. 4) specifies 
the AL-PDUs time coordination and required buffer size 
characteristics. The purpose is to ensure that AL-PDUs will 
arrive in a timely manner and conveying enough data to 
permit the data object to be fully updated for timely display. 
The AL-PDUs are stored in decode buffers, are reassembled 
into access units, are decoded (in accordance with the 
Decoder Type associated with the elementary stream), and 
stored in a configuration buffer, to be placed into the scene 
by the compositor which arranges the scene elements of the 
scene. DMIF also configures the multiplexing and access 
layer for each elementary stream based on its AL-PDU 
Configuration Descriptor. When DMIF is not used (typically 
in the case of a storage network), the association tag is 
substituted for a channel number identifying a logical or a 
physical channel containing the elementary stream. 

The process of sending a video stream begins with a 
moving picture request from the receiver application (in a 
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broadcast system the request goes no further than the 
receiver DMIF), followed by a sequence of data from a 
sender application designed to establish an initial scene 
configuration. In FIG. 4, one of the decoder types is for 
scene description and another is for object descriptors. In the 
initial scene configuration a number of object descriptors are 
typically established. Each data object descriptor includes at 
least one elementary stream descriptor. 

Another issue faced in an MPEG-4 system is avoiding 
collision among elementary stream identifiers when two 
applications run simultaneously on top of the same DMIF 
session. Each elementary stream ID is unique within an 
application. Therefore, the underlying DMIF session must 
be able to distinguish requests from each application in the 
case where elementary stream ID values collide. 

Hamaguchi U.S. Pat. No. 5,276,805 discloses an image 
filing system in which retrieval data for a first set of images 
is associated with image data for a first image that is 
logically associated with the first set of images. For 
example, a user viewing an x-ray image of a patient's kidney 
would be able to view a list of a set of x-ray images taken 
from the same patient and be able to quickly view any of the 
set of x-ray images upon request. Hamaguchi teaches that 
the retrieval information is associated with the image as a 
.whole. 

Judson U.S. Pat. No. 5,572,643 discusses an already 
existing technology, known as hypertext, by which an inter- 
net browser permits a user to access an internet URL by 
clicking on jL_highlighted wordj fl a page accessed on the 
internet. Juason Teiehes^mal The retrieval information is 
associated only with the test. Cohen et al. U.S. Pat. No. 
•5,367,621 is quite similar to Judson, permitting a user to 
click on a word in an on-line book, thereby causing a 
multimedia ob]ecTTo~lJe~diSplayed. Both Judson and Cohen 
et al. disclose advances linking together data which is 
unrelated to the encoding a nd da ta compression necessary 
for MPECPT aild MPEG- T 

SUMMARY OF THE INVENTION 

In a first embodiment, the present invention is a video 
system that includes a plurality of frames of video each of 
which is defined by a plurality of scene elements. The scene 
elements for a respective frame together define an image of 
the frame. First auxiliary data is descriptive of a first scene 
element of the frame and second auxiliary data is descriptive 
of a second scene element of the frame. A sending device 
sends the frame of video, including its scene elements, the 
first auxiliary data, and the second auxiliary data, from the 
sending device to a receiving device. 

Auxiliary data is data which does not directly represent 
audio or video but is at least one level of abstraction away 
from a direct audio or video representation or is information 
which is in some way related to the object. By including 
auxiliary data with the video system descriptive of the 
individual scene elements, in contrast to the entire frame, 
additional information specific to scene elements becomes 
available to the receiver. As an example, the additional 
information may be displayed on command or used to 
further process the video information. This information may 
or may not be synchronized to the data stream(s) conveying 
the scene element sequence. 

In another embodiment of the present invention the video 
system includes a plurality of frames of video, each of which 
defines an image. Each of the frames is defined by a plurality 
of pixels. First auxiliary data is descriptive of a first portion 
of the pixels of the frame and associated with a first portion 
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of the pixels. Second auxiliary data is descriptive of a second 
portion of the pixels of the frame and associated with a 
second portion of the pixels. A sending device sends the first 
portion of the pixels, the first auxiliary data, the second 
5 auxiliary data, and the second portion of the pixels to a 
receiving device. 

The inclusion of auxiliary data into video systems that 
define each frame by a group of pixels, such as MPEG-2, 
allows sub-portions of each frame to be associated with the 
10 auxiliary data for subsequent use. 

The foregoing and other objectives, features, and advan- 
tages of the invention will be more readily understood upon 
consideration of the following detailed description of the 
invention, taken in conjunction with the accompanying 
15 drawings. 

BRIEF DESCRIPTION OF THE SEVERAL 
VIEWS OF THE DRAWINGS 

20 FIG. 1A is an illustration of the data encoding and scene 
reconstruction of MPEG-4. 

FIG. IB is an illustration of a systems decoder for MPEG 

4. 

FIG. 2 is an illustration of the data structure of MPEG-4. 
25 FIG. 3 is an illustration of an MPEG-4 object descriptor. 
FIG. 4 is an illustration of an MPEG-4 elementary stream 
descriptor. 

FIG. 5 is an illustration of a receiver DMIF and a sender 
30 DMIF user interfaces according to the present invention. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENT 

When MPEG-4 was first conceived, it was anticipated 

35 that it would be a system for compressing and then subse- 
quently decompressing and displaying visual and audio data. 
This is in accordance with previous video compressing 
systems, such as MPEG-2. The present inventor came to the 
realization that other types of data that are specific to scene 

40 elements would be useful to the user and greatly expand the 
potential of an MPEG-4 system. For example, information 
concerning a visual object, such as the height of a building, 
~~ the make or model of an automobile, a basketball player's- 
statistics, or a link (pointer) to related information may be of 

45 interest to a viewer. Also, statistical information about the 
visual object, for instance a color description describing and 
quantifying the coloration of a particular object may be of 
use to the receiver in correctly displaying the object. In 
addition, data describing audio elements may also be 

50 included. This type of data is, in this application, referred to 
as "auxiliary data." Auxiliary data is data which does not 
directly represent audio or video but is at least one level of 
abstraction away from a direct audio or video representation 
or is information which is in some way related to the object. 

55 Auxiliary data can be used to enhance the original video/ 
audio service experienced by the viewers. 

In a preferred embodiment of the present invention, 
auxiliary data may be displayed, on command, in association 
with a scene element. For example, a person watching a 

60 basketball game may wish to learn more about any one of 
the players. Television sets, or other display systems, may be 
equipped with a mouse, a remote command unit, or other 
pointing device to select scene elements being displayed. 
Using such a device, a viewer could point to one of the 

65 basketball players, press a button, and thereby cause the 
players statistics to be displayed. Such auxiliary data may be 
transmitted in a simple ASCII format. The user may select 
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where the information is displayed, for example at either the to the receiver DMIF (step 128) and then processed by the 

top or bottom of the screen. Another use for auxiliary data receiver MPEG-4 synchronization layer 130 which reas- 

is for an on-screen program guide with brief program sembles the elementary stream as well as ensures timely 

descriptions. The brief program descriptions of the program prese ntation of the data to the application, 

guide provided may be selected to obtain a more complete 5 \fhi an alternative embodiment of the present invention, 

description of the programming. Yet another use of auxiliary MPEG-2 is equipped with a similar feature. The au xiliary 

data is to obtain additional information about an actor while data is inserted and multiplexed at the sending device into an 

viewing a television show. MPEG-2 transport stream. In one preferred embodiment, the 

Other uses for the auxiliary data could be in conjunction auxiliary at the sending device into an MPEG-2 program, 

with "hot buttons" displayed on the television screen. For 10 This is accomplished by assigning a special stream type 

example, there could be a hot button for a prewritten value specifying that the type of program element carried 

commentary on the show, or to allow the viewer to see and within the packets is auxiliary da ta^encoded according to a 

to add to a running commentary by viewers watching the particular format. Values oi sTrcfe-type have already been 

snow defined in Table 2-29 of the MPEG-2 systems specification 

The preferred embodiment of the present invention 15 (IEC/ISO 13818-1) which permit the announcement of 

includes an elementary stream descriptor, similar to that vWffl^,«i^ 

u • cio iu *• i.- i. j j , The stream type value assigned to the auxiliary data stream 

shown in FIG. 2 but in which one or more decoder type (or fa then t rans m itted,to the receiver by means of the TS 

equivalents stream type) structures announce the presence m map sections sT Hicture defm eU in Table 2-28. The 

of one or several auxiliary data streams encoded according auxiliary data stream(s) stream type and elementary PID 

to the format specified by decoder type. The elementary 20 va i ue (s) are added to the relevant program (defined by 

stream descriptor(s) including such decoder type (or stream program number in the TS program map sections structure), 

type) structure may be included in an object descriptor The inner descriptor loop in the TS program map section 

featuring elementary stream descriptor featuring other val- ma y be used in connection with the auxiliary data stream 

ues of decoder type (or stream type) like MPEG-4 visual type to provide additional, higtileve l, infuimal iolTaFout the 

streams. In this case the auxiliary data is associated with and 2 5 auxiliai^uiala-s^rvice. The TS program map sections allows 

complements the elementary streams published by these the transmissionandTeconstruction in the receiver of the 

elementary stream descriptors). In another situation, the program map table. The program map table provides the 

elementary stream descriptors announcing the presence of mapping between program numbers and the program ele- 

auxiliary data may be included in an object descriptor which m ents comprising them. The elementary PID value associ- 

does not feature elementary stream descriptors featuring 30 ated witn tne auxiliary data stream type specifies the packets 

other kinds of decoder type (or stream type). In this conveying the auxiliary data in the MPEG-2 transport 

situation, the auxiliary data may be associated and may stream. The auxiliary data is associated and enhances the 

complement a scene element 20 in a node 26 which is service P r0V1 <* ed b V other streams (video or audio elemen- 

directly linked to the node publishing the auxiliary data st ' eams > for example) published in the program map 

object descriptor. « table ' Each P r ?S ram ma * featur f. one or several auxiliary 

i, * . , data streams. In another embodiment, the - auxiliary data 

The present invention ensures that the elementary streams streams are announced and associated witrotherstreaTSs 

and the associated auxiliary data streams can be unambigu- fike_videiL^r audio elementary streams in an MPEG-2 

ously identified in the MPEG-4 player as a result of the program stream map structure (defined in Table 2-35 of 

DMIF assigning an association tag to each elementary ISO/IEC 13818-1). This is achieved by assigning a particu- 

stream. The association tag data is transmitted via a control 40 lar (stream type, elementary stream ID) tuple to announce a 

message from the sender DMIF and to the receiver DMIF. A particular auxiliary data stream format. Values of elementary 

separate channel handle uniquely representative of the use of stream ID have already been defined in Table 2-18 of 

a~n association tag by an application is assigned at run-time ISO/IEG 13818-1 which can be used to publish the presence . 

by the server and the client DMIF, respectively. Not only of video or audio streams in the multiplexed stream. The 

does this provide the receiving application with a shorthand 45 inner descriptor loop associated with the auxil iary data 

way of referencing the channel conveying the AL-PDUs, but stream type may be used to convey additionairfiigh--leveP 

it avoids confusion when at least two applications using the information about the auxiliary data service. In either 

same elementary stream identifier are running simulta- embodiments, and if the progranTineludesTeference to at 

neously in the same DMIF session. least one video elementary stream, a given auxiliary data 

rlfc. 5 illustrates the assignment of association tags and 50 Str ? am may be a ff ociat f onl y a P ortion of 

cnannel handles. The sender MPEG-4 application 110 sends miUe ^ Vlde ° P ° rtl ° t n K ^ch . portK \ n T y corres P° T nd * a 

♦u • •*• 1 u- ♦ j - * • 1 j- 1 . . specific region on the television display screen. In other 

the initial object descriptors finding elementary stream word ^mary data is associated witha particular block 

^T^Ll^T™ U2) 10 thC ****** DMIR ^ of picture elements (pixels) of a scene. Because o f encoding 

sender DMIF 114 during setup assigns an association tag mec hanism used by MPEG-2 video specification ^SO^Ec 

and a sender MPEG application program channel handle to 55 13818-2), it is preferable to define such picture block to 

each requested elementary stream (step 116) and passes the coincide with boundaries of a set of contiguous 16x16 pixel 

sender channel handle back to the sender MPEG-4 applica- blocks. Furthermore, if the data object to which t heauxjlia gi 

tion (step 117). The receiver DMIF 118 assigns a receiver dat a was associated at the sending device was mjwin ga5ou t 

channel handle to each requested elementary stream (step thVsefee fl, th e auxil iary-data'may'be reassigned to pixel se ts_ 

120) and passes it (step 122) to the receiver MPEG-4 60 on^bstantialIy^a ^ame=togtonic"or field-to-field ba sisj^"" 

application 124, thus creating an end-to-end pipe indicator "*Tfre*1erms and expressions which have been employedln 

composed of the sender channel handle, association tag, and the foregoing specification are used therein as terms of 

receiver-channel handle. When elementary stream data is description and not of limitation, and there is no intention, 

sent from the sender MPEG-4 application 110 to the sender in the use of such terms and expressions, of excluding 

DMIF 114, it is first processed by the MPEG-4 synchroni- 65 equivalents of the features shown and described or portions 

zation layer 140. The packetized, possibly time stamped, thereof, it being recognized that the scope of the invention 

possibly multiplexed elementary stream is then transmitted is defined and limited only by the claims which follow. 
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What is claimed is: 

1. A video system comprising: 

(a) a plurality of frames of video each of which is defined 
by a plurality of scene elements, where said plurality of 
scene elements for a respective frame together define 
an image of said respective frame, wherein each of said 
scene elements is a separately encoded image with 
respect to other scene elements for said respective 
frame, wherein said image is created by composing 
said separately encoded scene elements within a 
respective frame based upon location information 
defining the position of said scene elements within said 
respective frame by a receiving device of said video 
system; 

(b) first auxiliary data descriptive of the content of a first 
scene element of said plurality of scene elements for 
said respective frame; 

(c) second auxiliary data descriptive of the content of a 
second scene element of said plurality of scene ele- 
ments for said respective frame; and 

(d) a sending device that sends a bitstream representing 
said respective frame of said video, including respec- 
tive said separately encoded scene elements and said 
location information, said first auxiliary data, and said 
second auxiliary data, from said sending device to said 
receiving device, wherein said separately encoded 
scene elements are sent from said sending device to 
said receiving device using at least two elementary 
streams. 

2. The video system of claim 1 wherein said video 
includes audio information. 

3. The video system of claim 1 wherein said scene 
elements are overlapping. 

4. The video system of claim 1 wherein said first auxiliary 
data is a set of descriptors and parameter values resulting 
from said first scene element. 

5. The video system of claim 1 wherein said first auxiliary 
data is not directly representative of an image. 

6. The video system of claim 1 wherein said first auxiliary 
data is viewable on a display device by selecting said first 
scene element. 

7. A video system comprising: 

(a) a plurality„of frames of video. each of which defines an 
image of said respective frame, each of said frames 
defined by a plurality of pixels; 

(b) first auxiliary data descriptive of the content of a first 
portion of said pixels of a respective frame and asso- 
ciated with said first portion of said pixels of a respec- 
tive frame and associated with said first portion of said 
pixels; 

(c) second auxiliary data descriptive of the content of a 
second portion of said pixels of said respective frame 
and associated with said second portion of said pixels, 
where said first portion is different than said second 
portion; and 

(d) a sending device that sends said first portion of said 
pixels, said first auxiliary data, said second auxiliary 
data, and said second portion of said pixels to a 
receiving device. 

8. The video system of claim 7 wherein said video 
includes audio information. 

9. The video system of claim 7 wherein said first portion 
and said second portion are blocks of 16 by 16 pixels of said 
respective frame. 

10. The video system of claim 7 wherein said first 
auxiliary data is a set of descriptors and parameter values 
resulting form said first portion. 
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11. The video system of claim 7 wherein said first 
auxiliary data is not directly representative of an image. 

12. The video system of claim 7 wherein said first 
auxiliary data is viewable on a display device by selecting 

5 said first portion. 

13. A video system comprising: 

(a) a plurality of frames of video each of which is defined 
by a plurality of scene elements, where said plurality of 
scene elements for a respective frame together define 

10 an image of said respective frame, wherein each of said 
scene elements is a separately encoded image with 
respect to other scene elements for said respective 
frame, wherein said image is created by composing 
said separately encoded scene elements within a 

15 respective frame based upon location information 
defining the position of said scene elements within said 
respective frame by said video system; 

(b) each of said scene elements defined by a scene 
description format including a scene element data 

20 object format that defines the number of elementary 
streams and elementary stream descriptors; 

(c) each of said elementary stream descriptors including a 
decoder type identifier; 

25 (d) each of said decoder type identifiers represents at least 
one of an audio stream, a video stream, object 
descriptor, and auxiliary data, with at least one of said 
decoder type identifiers represents said auxiliary data; 
and 

30 (e) said auxiliary data descriptive of the content of a 
respective one of said scene elements. 

14. A system for encoding a moving picture comprised of 
a sequence of time separated frames depicting scenes, 
wherein each of said frames is defined by a plurality of scene 

35 elements, wherein each of the scene elements is a separately 
encoded image with respect to other scene elements for the 
respective frame, wherein said image is created by decoding 
a bitstream and composing said separately encoded scene 
elements within a respective frame based upon location 

40 information defining the position of said scene elements 
within said respective frame, said system comprising a scene 
description format, referencing at least one of said scene 
- elements, including ^ scene element data object format-that 
includes an elementary stream descriptor format including, 

45 in turn, a decoder type identifier format wherein any one of 
a set of decoder types may be represented by one out of a set 
of p redetermin ed bi t^ values and one of said predetermined 
biTyalueja. represents an auxiliary information decoder type, 
wherein said auxiliary information is descriptive of the 

50 content of at least one of said scene elements. 

15. The system of claim 14 wherein an audio stream is 
also encoded. 

16. A system for encoding a moving picture comprised of 
a sequence of lime separated frames, wherein each of said 

55 frames includes a plurality of scene elements that are a 
separately encoded image with respect to other scene ele- 
ments for a respective frame, said system adapted to divide 
said frames into scene elements, wherein said frame is 
created by composing said scene elements within said 

60 respective frame based upon location information defining 
the position of said scene elements within said respective 
frame by said system, and including a receiver display 
subsystem adapted to receive and display said frames and a 
pipe assignment subsystem adapted assign pipes to the scene 

65 elements and wherein said subsystem assigns at least one of 
a logical and physical transmission channel designator to 
each pipe and transmits a quantity representative of said pipe 
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designator to said receiver display subsystem, wherein said 
system includes a plurality of auxiliary data descriptive of 
the content of a respective plurality of scene elements. 

17. The system of claim 16 wherein an audio stream is 
also encoded. 
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18. The system of claim 13 wherein said plurality of 
frames including said scene elements are represented in a 
bitstream. 

* * * * * 
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